Crate selium_log

A library containing an implementation of an ordered, log-based message queue.

Selium Log aims to provide as simple an abstraction as possible over the message log, in order to make it easy to provision and use in your libraries.

The message log’s structure should be familiar to those coming from an Apache Kafka background: a log is represented by one or more segments, with each segment comprised of a memory-mapped index file, serving as a lookup for the segment’s records, and an append-only data file containing said records.

Only the most current segment can be designated as the mutable (“hot”) segment, while the older segments are read-only until their eventual cleanup. Once the current hot segment exceeds the defined LogConfig::max_index_entries threshold, it will also become read-only, while a new segment is created and assigned as the hot segment.

Replication has not yet been implemented for Selium Log as of this release, but is a planned feature. Due to this, durability can be tough to achieve alongside high throughput on a single node. Most of the latency comes from the I/O overhead of flushing the memory-mapped index and data files to the filesystem. To compensate for this overhead, the flushing frequency can be tweaked via the FlushPolicy struct, to strike a balance between durability and throughput.

Modules

• config
  Configuration settings for an individual message log.
• data
  Contains the Data type, along with a LogIterator to iterate over the data file.
• error
  Type aliases and Enums relating to Selium Log errors.
• index
  Contains the Index and Mmap types.
• message
  Message envelope/frame for a set of records.
• segment
  Contains the Segment and SegmentList types.

Structs

• MessageLog
  The entry point to Selium Log.